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Abstract. The q, ^-Catalan numbers can be denned using rational functions, geometry re- 
lated to Hilbert schemes, symmetric functions, representation theory, Dyck paths, partition 
statistics, or Dyck words. After decades of intensive study, it was eventually proved that 
all these definitions are equivalent. In this paper, we study the similar situation for higher 
q, t-Catalan numbers. We compute the limits of several versions of the modified higher 
q, t-Catalan numbers and show that these limits equal the generating function for integer 
partitions. This result supports the conjecture that the combinatorial definitions of higher 
q, t-Catalan numbers are indeed equivalent to the algebraic and geometric definitions. 



1. Introduction 

The purpose of this paper is to study the different definitions of higher q, i- Catalan 
numbers and provide support to the conjecture that they are indeed equivalent. We begin 
by reviewing seven ways of defining the higher q, t-Catalan numbers in (a)-(g) below. We 
then indicate which of these definitions are already known to be equivalent. 




mn, n) 



(0,0) 

Figure 1. Definition of l(x), a(x), l'(x), and a'(x). 



(a) Suppose A is an integer partition with Ferrers diagram dg(A). Define area(A) = |A| = 
| dg(A)|, the number of cells in the diagram of A. For a cell x G dg(A), define the leg l(x), 
the arm a(x), the coleg l'(x), and the coarm a'(x) to be the distances shown in Figured] 
Let Par^" 1 -* be the set of partitions A such that dg(A) fits in the triangle with vertices (0, 0), 
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(0, n), and (mn, n) when drawn as shown in the figure. For such partitions, define area c (A) 
m^) ~ area(A) and 

c m (A) = \{x G dg(A) : ml(x) < a(x) < ml(x) + m}\. 
For example, when m = 2 and A = (7, 5, 4), we have |A| = 16 and 02(A) = 13. 
Define the partition version of the higher q,t- Catalan numbers by 

PC { ™\q,t)= g areaC(A) t Cm(A) . 

AeParl m) 

For example, PCf\q, t) = q 3 + q 2 t + qt 2 + t 3 and 

PCf\q, t) = q e + qh + qH 2 + qH + q 3 t 3 + q 3 t 2 + q 2 t 2 + q 2 t 3 + gt 4 + q 2 t 4 + qt 5 + t 6 . 



(b) An m-Dyck word is a sequence 7 = (70, 71, ... , 7 n -i) such that % 6 N = {0, 1, 2, . . .}, 
70 = 0, and 7 m < j { + m f« 
length n. For such a word ^ 
Eo<«<i< n sc ™(7i - 7i), wh ere 



7o = 0, and 7 i+ i <7j + mfor0<i < n — 1. Let ri m ' ) be the set of m-Dyck words of 
length n. For such a word 7, define area(7) = Y^=o 7»- ^ s m define dinv m (7) = 



- p, if 1 < p < m 
sc m (p) = ^ m + p, if —?77 < p < 

for all other p. 




(2) 

For example, 7 = (0, 2, 0, 1, 1) G I5 has area(7) = 4 and dinv 2 (7) = 13. 
Define the word version of the higher q,t-Catalan numbers by 

WC { ™\q,t)= g arca(7) t dinVm{7) . 

(c) An m-Dyck path of order n is a lattice path 7r from (0, 0) to (mn, n) using north and 
east steps such that the path never goes below the diagonal line segment with endpoints 
(0,0) and (mn,n). Let be the set of such m-Dyck paths. For such a path tt, let 

area(7r) be the number of complete unit squares between tc and the diagonal. Define the 
m-bounce statistic 6 m (7r) as follows. Set Vi — for all negative integers i. Starting from 
(0, 0), construct a bounce path by induction on i > 0. In the (i + l)th step, move north from 
the current position (u, v) until hitting an east step of the m-Dyck path that starts on the 
line x — u, and define the distance traveled to be i>j. Then move east from this position 
Vi + v j_i + ■ — h v i- m +i units. Continue bouncing until reaching (mn, n). (In fact, it suffices 
to stop once we reach the horizontal line y = n.) Then b m (Tr) = ^2 k>0 kv k . For example, the 

path 7r G T>f^ in Figure H] has area(7r) = 4, (t> , vi, . . . , v 5 ) = (2, 0, 1, 1, 1, 0), and b 2 ('n) = 9. 
Define the Dyck path version of the higher q,t-Catalan numbers by 

DC^\q,t)= g M7r) t areaW . 
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Figure 2. An m-Dyck path. 



(d) The q, t-Catalan numbers may be defined using symmetric functions, as follows. This 
discussion assumes the reader is familiar with the elementary symmetric functions e n , the 
modified Macdonald polynomials H^, and the Hall scalar product (•, •) on symmetric func- 
tions; see [I] or [6j §3.5.5] for details. For any integer partition /i, define n(/z) = XLgdgO) K x ) 



and n(fi') = ^2 



xgdgO') 



a(x), where /i' denotes the transpose of \i. Define 

Tfi = q n ^'H n ^\ Let A denote the ring of symmetric functions with coefficients in the field 
F = Q(q,t). The Bergeron-Garsia nabla operator [1] is the unique F-linear map V on A 
that acts on the modified Macdonald basis via V(i/ M ) = T^H^ for all partitions fi. For 
m G N + = {1, 2, 3, . . .}, V" 1 denotes the composition of m copies of the operator V. We now 
define the symmetric function version of the higher q,t- Catalan numbers by 

SCi m \q,t) = (V m (e n ),e n ). 



(e) The higher q, i-Catalan numbers were originally defined by Garsia and Haiman in [3] as 
sums of rational functions in Q(q, t) constructed from integer partitions. Recall that /ihn 
means that \i is an integer partition of n. With T M defined as in (d), we further define 

b,= <f' {x) t v{x \ n M = J] (i-<fW^), 

xedg(At) xedgOu)\{(Q,o)} 
WfM = Yl [(q a{x) -t l W +1 )(t l(x) -q ai - x)+l )}. 

xGdg(/i) 

Then the rational function version of the higher q,t-Catalan numbers is defined by 

RCir\q,t) = £(1 - q)(l - t)T™ +1 B,Ujw,. 

Iihn 

(f) For fixed n G N + , consider the polynomial ring C[x, y] = C[xi,yi, ■ ■ ■ , x n , y n ). S n acts 
diagonally on this ring by the rule w ■ Xi = x w ^), w ■ yi = y w {i) for w G S n and 1 < i < n. 
A polynomial / G C[x, y] is called alternating iff w ■ f — sgn(w)/ for all w G S n . Let I 
be the ideal in C[x, y] generated by all alternating polynomials, and let m be the maximal 
ideal generated by X\, yi, . . . , x n , y n . We write I = I n and m = m n if it is necessary to 
indicate the number of variables. Let = I m /ml m for m G N, and for simplicity, 
let M = M^. Given a monomial / = x^y^ 1 ■ ■ ■ x^y^ 1 G C[x, y], we define the bidegree 
of f to be the ordered pair (X)^ =1 a*, Yli=i h)- We say that a polynomial in C[x, y] is 
bihomogeneous of bidegree (d±, d?) if all its monomials have the same bidegree (di, d^). Then 
I m and M*™ - ) become doubly-graded S^-modules by taking bidegrees in the x-variables and 
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the variables. Let Mi™ denote the bihomogeneous component of of bidegree (it, v). 
Define the algebraic version of the higher q,t- Catalan numbers by 

AC™ (q, t) = J2J2 dim M ff • 

«>0 v>0 

For more information, see [31 Section 3]. 

(g) Finally we state the geometric definition (see [8j [10] for more details). Let Z n be the 
zero fiber of the Hilbert-Chow morphism Hilb n (C 2 ) — > Sym n (C 2 ), let 0(1) be the restriction 
of the ample line bundle on Hilb n (C 2 ) induced by the isomorphism Hilb n (C 2 ) = Proj(T), 
where T = @ d>Q A d and A = C[xi,yi, . . . ,x n ,y n ] e is the space of ^-alternating elements. 
For any m G N+, let 0{m) = C(l)® m . The set of global sections H°{Z n , 0{m)) is a bigraded 
vector space. Define the geometric version of the higher q,t- Catalan numbers by 

GCj™\q,t) = J2<l U t V &™H (Z n ,O(m)) UiV . 



It is conjectured that the seven definitions (a)-(g) of higher q, t-Catalan numbers are all 
equivalent. This conjecture is supported by explicit computations for small values of m and 
n. It has been proved that for all m, n e N + , 

(1.1) PCl m \q,t) = WCl m \q,t) = DCi m \q,t) and 

(1.2) SCi m \q,t) = RC<r>(q,t) = AC^\q,t) = GC^ (q, t) . 

We discuss the proofs of these equalities in an appendix (§7). It remains to be proved that 
the three combinatorial definitions agree with the four algebraic and geometric definitions. 
This conjecture has already been proved for certain specializations of the parameters q and 
t. For instance, using [21 Theorem 4.4] and definitions (a) and (c) above, we find that 

RClr\q,l) = RCi m \l,q)= ]T q^ = PCl m \q,l) = DCl m \l,q). 

Upon setting t = 1/q, we see from [31 Corollary 4.1] and [121 §3-3] that 
RCt\q, l/q)q m ^ = WC^\q, l/q)q m ® = DC^(q, l/q)q m ^ = 

This paper provides more evidence of the conjectured equality of the seven definitions 
of higher q, t-Catalan numbers. Specifically, we will study the limiting behavior, as n tends 
to infinity, of the "modified" higher q, t-Catalan numbers given by 

q m ^PCi m \q-\t) } q^DCMfaq- 1 ), and q m ® AC™ (q- 1 , t) . 

We will show that all of these polynomials have as their limit the famous generating function 
n^i(i — tq l )~ l , which enumerates integer partitions by area and number of parts. (Here we 
are taking limits in a formal power series ring, which means that for each fixed monomial 
q a t b , the coefficient of this monomial becomes stable for sufficiently large n.) We state our 
main theorem more precisely as follows. 



mn + n 
n 
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Theorem 1.1. For any positive integer m, we have 

lim q^PC^iq-^t) = lim q m ti) DC^ (t, q' 1 ) = lim q m &) AC™ {q~\t) 



n— >oo 

oo 



i=l /igPar 

where Par is the set of all integer partitions, and £(fi) is the number of parts of p,. 

The paper is organized as follows. In §2 we introduce further notation, background, and 
preliminary results. In §3 we prove the combinatorial part of our main theorem. In §4 we 
prove the algebraic part of the main theorem. In §5, we extend the method used in the proof 
of the main theorem to identify certain coefficients of AC ( ^ n \q^t) as partition numbers. In 
§6 we give some related conjectures. In §7 we indicate the proofs of the equalities stated 
in ([Li]) and (Tl~2]) . 



2. Background and Notation 



2.1. Notation. 



• For k, b G N + , denote the set of integer partitions of k by Par(fc), and denote the 
set of integer partitions of k into at most b parts by Par (b, k). More explicitly, 

Par(/c) — {v — (z/i, z/ 2 , . . . , v t ) \ v { G N + , v x < v 2 < ■ ■ • < v e , v x + v 2 H h v t — k~) 

and Par(6, k) — {y — (z/ 1; z/ 2 , . . . , Ui) G Par(/c)| £ < b}. By convention, Par(O) = {0}, 
Par(0, k) = for k > 0, and Par(/i, 0) = {0} for all h > (where {0} is a set 
with one element). Let p{k) and p(b, k) be the cardinalities of Par(/c) and Par (6, k), 
respectively. In other words, p(k) is the number of partitions of k and p(h, k) is the 
number of partitions of k into at most h parts. By the above conventions, p(0) = 1, 
p(0, k) = for k > 0, and p(h, 0) = 1 for all h > 0. 

• Let C[p] = C[pi, p2, ■ ■ ■ ] be the polynomial ring with countably many variables pi, for 
i G N + . As a convention, we set po = 1. For a partition v = (yi, z/2, ... , ug) G Par(/c), 
define p u = p ul Pu 2 ' ' ' Pu t ^ Define the weight of a monomial cp^ • • • p i( (where 
c G C \ {0}) to be %x + • • ■ + it- For w G N, define C[p\ w to be the subspace of 
C[p] spanned by monomials of weight w. For / G C[p\, there is a unique expression 

/ = Y^w^if}™ witn {f}™ G and we cal1 if}w the weight-w part of /. 

• For P = (a, b) G N x N, we write \P\ = a + b, \P\ X = a, and \P\ y = b. 

• For n G N + , define 

5) n = {£> c N x N : \D\ =n}. 

For D G D„, we write D = {Pi, P 2 , • • • , P n } where each P = (a i? &,) G N x N. Unless 
otherwise specified, we always choose notation so that Pi, . . . ,P n are in increasing 
graded lexicographic order. This means that Pi < Pj < ■ • ■ < P n , where 

(a, b) < (a, b') if a + b < a' + 6', or if a + b = a + b' and a < a' . 

To visualize a set D G we can draw a square grid on which we plot the n ordered 
pairs in D. For example, in the following picture, the horizontal and vertical bold 
lines represent the x-axis and y-axis, and D = {(0, 0), (1, 0), (1, 1), (2, 0), (3, 0)}. 
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m m 



Given D = {Pi, . . . , P n } G D n , define the total degree, x-degree, y-degree, and bidegree 

of d to be YTi=i (\Pi\x + \Pi\y), Er=i Er=i l p 4 ; and (E£=i l p ;UEr=i l p < 



respectively. Then the x-degree (resp. y-degree) of D will be denoted by di(D) (resp. 
d 2 (D)). Let k(D) = Q) - d x {D) - d 2 {D). 

The diagonal ideal / of C[x, y] and the bigraded C- vector space M = ® d dagN M dljd2 
were defined in §1 (f ) . The ideal generated by all homogeneous elements in / of total 
degree less than d is denoted by I <( i. 

For D = {(ai, bi), (a n , b n )} G D n , the alternating polynomial A(D) G C[x, y] is 
defined by 



A(D) = detlx^y^iKijKn = det 



rf ,a\ 1 b\ 



™12 ^2 
2^1 2/1 



X l ill 



""n an 



Note that A(D) is bihomogeneous of bidegree equal to the bidegree of D. 
Given two polynomials f,g G I m of the same bidegree (di,d 2 ), let f,g be the corre- 
sponding elements in M^ 1 ^. For m — 1, we say that 

/ = g (modulo lower degrees) 

if / = g in M d d2 , or, equivalently, if / - g is in I <dl +d 2 - 



2.2. Properties of the module M dljd2 . To be self-contained, we review a few definitions 
given in [11]. The reader is suggested to look at Example 12.41 while reading these definitions. 

Definition-Proposition 2.1 ([HI Definition-Proposition 6]). Let D = {Pi, . . . , P n } G D n , 

and write = (a iy bj). Then there is an n x n matrix S whose (i, j)-entry is 

JO, if % < \Pj\; 

\ Ziiz i2 ■ ■ ■ £i,|Pj| where zu is either Xi — xe or y^ — y^, otherwise, 

for all 1 < i,j < n, such that det(S') = A(P) (modulo lower degrees). We call S a staircase 
form of D. 

Definition 2.2. Let D and S be defined as in Definition-Proposition I2TT1 Consider the set 
{j : \Pj\ = j — 1} = {ri < r2 < • ■ • < r^} and define r^ + i = n + 1. For 1 < t < £, define 
the t-th block B t of S to be the square submatrix of S of size {r t +\ — r t ) whose upper-left 
corner is the (r t , r 4 )-entry. Define the block diagonal form B(S) of S to be the block diagonal 
matrix diag(Pi, . . . , Bi). 

Definition 2.3. Let S be a staircase form, B(S) be its block diagonal form with blocks 
Pi, ... , Bg. For 1 < t < £, let ^ be the number of nonzero entries in block B t that are strictly 
above the diagonal, i.e., the number of nonzero i, j-entries in B t where j > i. Eliminating 
zeros in /^) and then rearranging the sequence in ascending order, we obtain a 

partition of k, denoted by (J>(S). We say that S is of partition type (i(S). We call a block B t 
minimal if every (i, j)-entry (j > i + 1) that lies in B t is zero. We call S a minimal staircase 



LIMITS OF MODIFIED HIGHER q, i-CATALAN NUMBERS 



7 



form if all the blocks in B(S) are minimal. We say D G D n is of partition type )J-(S) if S is 
a staircase form of D. (Note that the partition type does not depend on the choice of S.) 

Example 2.4. (i) Let D = {(0, 0), (0, 1), (0, 2), (1, 1)} G D 4 . We list here A(D) and a 
possible staircase form S together with the corresponding block diagonal forms B(S). In 
this example, D is of partition type (1). 
blocks of size 1 and one block of size 2. 



S is a minimal staircase form, and B(S) has two 



A(D) 



where x 



Vi 

y-2 

y-i 



vi 
vi 
vi 
i 



x iVi 

X 2V2 

x 3 y 3 

£42/4 



S 





2/21 
2/31 
2/41 






2/312/32 
2/412/42 






^312/32 
£412/42 





2/21 









2/312/32 
2/412/42 






£312/32 
£412/42 



and y y = yi - yj. 



(ii) Let D = {(0, 0), (0, 1), (1, 0), (1, 2), (2, 1), (3, 0)} G S 6 . A staircase form S and the 
corresponding block diagonal form B(S) are given below. Then D is of partition type (1, 3), 
B(S) has three blocks of sizes 1,2,3 respectively, and S is not a minimal staircase form 
(because the (4, 6)-entry of B(S) is £41X42X43 7^ 0, therefore the block B 3 is not minimal). 





y-2i 

2/31 
2/41 
1/51 
2/61 





121 
Z31 
X41 
^51 
X61 







2/412/42X43 
2/512/52^53 
2/612/62X63 







2/41X42X43 
2/51^52^53 
2/612:62X63 







X41X42X43 
£51252253 
X61X62X63. 



B(S) 




2/21 
2/31 











X21 
^31 











2/412/42X43 
2/512/52^53 
2/612/62X63 







2/41X42X43 
2/51^52^53 
2/61^62X63 







X41X42X43 
^51^52^53 
X61X62X63. 



Theorem 2.5 ([HI Theorem 5]). Let n be a positive integer, and let di, d 2 , k be non-negative 
integers such that k = (L 1 ) — di — d 2 . Define 5 = min(di, d 2 ). Then dim Md lt d 2 — p($, k), and 
equality holds here if and only if either k < n — 3, or k = n — 2 and 6=1, or 5 = 0. 



Recall some definitions in For b G N and w G Z, define 

fc(6,w)= {(l + Pl +p 2 + ---) 6 } w - 
For D = {P u . . . , P n } G £> n , define y>(£>) to be 



[-l) k ^ det 



^6i,-|Pi|) ^l-lPil) ^,2-lPxl) 
h(b 2 ,-\P 2 \) h(b 2 ,l-\P 2 \) h(b 2 ,2-\P 2 \) 

h{b n ,-\P n \) h{b n ,l-\P n \) h{b n ,2-\P n \) 



h(bi,n 
h(b 2 ,n 



\Pi\) 

\P2\) 



h(b n ,n- 1 - \P n \) 



It is proved in [TTl Lemma 47] that <p induces a well-defined linear map ip : a 2 ~>* 
^[p]( n )-di-d 2 ' ^ e nave conjectured that </? is injective and proved the injectivity under the 
condition (™) — di — d 2 < n — 3 and d 2 < d\ [HI Conjecture 48, Theorem 43, Theorem 
44]. With a slight modification, we can prove the injectivity under the sole condition ("J — 
d\ — d 2 < n — 3 without the constraint d 2 < d\. (We briefly explain the modification using 
the terminology in [11] : assume now d 2 > d\. It suffices to prove that, for each v G Hdx,k, 
there exists an alternating polynomial g v such that the leading monomial lm(<£>(<7„)) = p u . 
In fact, such a g v can be obtained, up to a sign, by switching x- and y- coordinates of the 
f v constructed for M^di in [HI Theorem 44].) 
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Lemma 2.6. Suppose < ( n 2 — d[ — d 2 < n — A, d[ < d\, d' 2 < d 2 , and d[ + d' 2 + (n — 1) = 
d\ + d 2 . Let M' d i d , and M dl;d2 be the indicated bigraded components of J n _ 1 /m n _i/ n _i and 
In/w-nln, respectively. Let 

di—dty n-l 

fo = JJ (%n ~Xi)- Yl (f" ~ Vi)' 

i=l i=di-4+l 

TTien £/ie linear map h : M^, d , — > M dlyd2 that maps f to fof is injective. 

Proof. First observe that ( n 2 1 ) — d[ — d 2 = f") — d\ — d 2 , which we denote by k. It is also 
easy to check that h is well-defined. 

We now explain that the following triangle is commutative: 

M' d , 4 ,^UM dud2 

C[p] k 

i.e., <p'(f) is identical with <f(fof)- Indeed, for D' G £> n -i of bidegree (d±, d 2 ), let / = A(D'), 
then /o/ = A{D) for P> = £>' U {(d x - d[, n - 1 - d l + d[)} G D„. Then k(D') = k{D) = k. 
Let A' (resp. A) be the (n — 1) x (n — 1) matrix (resp. n x n matrix) in the definition of 
<p'(f) (resp. tp(fof))- Since A' is the first (n — 1) x (n — 1) minor of A and the last row of A is 
(0, . . . , 0, 1), det(A') = det(A). Therefore (p'{f) = (-l) fc det(A') = (-l) fc det(A) = <p(fof). 

Now since <p' : M' d , d , — > C [/?]*. is injective for < (n — 1) — 3, /i is also injective. □ 

Definition 2.7. Given d\ + d 2 = (™), take arbitrary P = {Pi, . . . , P n } G D n of bidegree 
(di,c?2) such that \Pi\ — i — 1. Define f dl ,d 2 to De * ne equivalence class of A(D) in M dll d 2 - 
By [HI Lemma 16], this equivalence class is independent of the choice of D. 

3. Limits of the Modified Combinatorial Higher ^-Catalan Numbers 

We recall the following theorem [T3l Thm. 3]. 

Theorem 3.1. For X G Par and m G M + ; define h^X) to be the number of cells x G dg(A) 
such that m^+i <m< "^l" 1 . Then 

? area(A)^+(A) = "Q 1 _ ? area(»^(/i) _ 

AGPar i=l ^ ^ePar 

Proposition 3.2. (i) Por a// m, n G N + , 

g^PaW^ 1 ,^ ^ g area(A) fCm (A)_ 

AGParl m) 
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(ii) For all m G N + , 

oo 1 

lim q m ^PC^\q-\t) = V f ca(A) t Cm(A) = TT : 

n-^oo ^ — ' - LJ - 1 — tc/ 1 

AePar i=l ^ 



Proof, (i) is straightforward. For (ii), if we increase n by 1, a partition A in the mn x n 
triangle will also fit into the m(n + 1) x (n + 1) triangle, and the two statistics area(A) and 
c m (A) do not change with n. Since all integer partitions of a fixed area will fit in the triangle 
for sufficiently large n, the first equality follows from (i). The second equality follows from 
Theorem 13 . 1 1 and the observation that /i+(A) = c m (A). □ 

Proposition 3.3. (i) For all m,n G N + , 

q m (^DCi m \t,q- 1 )= g areaCW t bmW , 

where area c (7r) is the number of lattice squares in the mn x n triangle above ir. 
(ii) For all m G N+ ; 

oo 1 

lim q^DC^it^- 1 ) = V g—W^W = TT r . 

n— »oo " ^ — ' J- J- 1 — to 1 

AePar i=l ^ 



Proof, (i) is straightforward. For (ii), note that each m-Dyck path tc G T>n determines an 
integer partition A = X(ir) whose diagram consists of the squares above ir in the mn x n 
triangle. For each n, there is an injection — > T>^ 1 that adds one north step to the 
beginning of tt and adds m east steps to the end of ir. This injection preserves both \(ir) and 
area c (7r) = area(A(7r)), but the value of the bounce statistic b m (ir) may change. However, 
as n continues to increase, the bounce statistic will eventually stabilize. More specifically, 
once n > 2area c (7r), it is routine to check that the bounce path will satisfy v o = n — £(X(ir)), 
V\ = £(X(tt)), and v , = for all i > 2; the key observation is that the first horizontal move (of 
length t> ) moves to the right of all the squares in dg(A(7r)). It follows that 6 m (vr) = £(X(7r)) 
for such n. By fixing the area of the partition outside 7r and taking n larger than twice this 
area, we see as in the previous proposition that the indicated limit holds. □ 



4. Limits of the Modified Algebraic Higher q, ^-Catalan Numbers 

Lemma 4.1. Let D G D n , let S be a staircase form of D, and let B(S) be the block diagonal 
form of S . Then the number of 1 x 1 blocks in B(S) is at least n — 2k(D). 

Proof. Suppose the number of size-1 blocks in B(S) is t, and the other blocks have sizes 
si, . . . , s r . On one hand, t + YH=i s * = n - On the other hand, a block of size Sj contributes 
at least Sj — 1 to k(D), hence ^i=i( s » ~~ 1) — k(D). Since Si > 2, we have s« < 2(sj — 1) and 
* = n - E!=i * > « - ELi 2 ( s < - !) >n-2k(D). □ 
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Lemma 4.2 ([HJ Lemma 15]). Let D = {Pi, . . . , P n } G D n and Pi = (dj, 6j) 6e as m §£. Lei 

j 6e £u>o integers satisfying 1 < i ^ j < n, \Pj\ = i — 1, \Pi+i \ = i, \Pj\ = j — 1, \Pj+i\ = j , 
hi > 0, aj > (we define |P n +i| = n). Let D' be obtained from D by moving Pi to southeast 
and Pj to northwest, i.e., 

D' — {Pi, . . . , Pj_i, Pi + (1, — 1), P+i, ■ • ■ , Pj-l, Pj + ( — 1, 1), Pj+l, ■ ■ ■ i Pn}- 

Then A(D) = A(D') (modulo lower degrees). 

Theorem 4.3. Assume n, m, k, d\, d 2 G N satisfy n > 3 , m > , k = —d\—d 2 < n/2—1, 
and d 2 < n/2 — 1. For each fi G Par(c?2, k), let be an arbitrary minimal staircase form 
of bidegree (d\ — (m — l)(^),d 2 ) and partition type /i. Then M^L is generated as a vector 
space by 

l(det^) \{ (xi-x.r-'l 

I l<i<j<n ) (LtePar(d2>fe ) 

Consequently, 

dimM^ 2 <p(d2,k). 

Proof. We use induction on m. The base case m = 1 is done in [TT] . Let us briefly sketch a 
proof for the base case. 

For each /i G Par(c?2, k), the assumption < n/2 — 1 implies that there exists a minimal 
staircase form, say S^, of bidegree (di — (m — l)^); d 2 ) and partition type /i. Let _D M be an 
element in D n , whose staircase form is S^. Since (p(A(D fl )) = and (p is injective in this 
case, M dl)d2 is generated by A(D Ai )(= det S^). 

Now assume that m > 2. Note that M^"^ is generated by products Y^L 1 A(D i ) 1 where 
Di G D n , Y^iLidi{Dj) = d\ and YliLi^iDi) = d 2 . So we only need to prove that each 
such product is a linear combination of {det(S' M ) Yli<i<j< n ( x i ~ x j)" 1 ' 1 } ^&a.r{d 2 ,k) modulo 
lower degrees. Define k! = k(Di), d' 2 = d 2 (Di), k" = k — k', d 2 = d 2 — d' 2 . By inductive 
assumption, Y\™= 2 A(.D,) is a linear combination of {det(5^') Yli<i<j< n ( 

modulo lower degrees, and A(L>i) is a linear combination of {det(S'^)} I/e p ar ( f j' modulo 
lower degrees. Hence ni^i A(-Dj) is a linear combination of {det(5^) det(S'^) Yli<i<j< n ( x i — 
x j) m ~ 2 }vePnr(d' 2 ,k'), AePar(4',fc") modulo lower degrees. So to prove the theorem, it suffices to 
show the following statement: 

(*) det(S^) det(S") is a linear combination of {det(5 At ) Yli<i<j<S x i -^j))>sPar(<i 2 ,fc) modulo 
lower degrees. 

Since S' u and S" can be arbitrary minimal staircase forms of fixed bidegree and fixed 
partition type, we may assume that all the lxl blocks but the first one in the block diagonal 
form B(S' V ) are below bigger blocks, and that all the lxl blocks in the block diagonal form 
B>(S") are above bigger blocks. Let T' (resp. T") be the product of determinants of the 
blocks of size greater than 1 in the block diagonal form B(S' U ) (resp. B(S")). We have 

n j-l / b j-1 \ 

det(sD = r n n %\ det(^) = n ikm ^ 

j=a i=l \j=2 i=l / 
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where z$ = Xi — Xj or yi — yj for t = 1,2. The numbers of size-1 blocks in B(S' U ) and B(S") 
are n — a + 2 and b, respectively. We assume without loss of generality that S' u has no more 
size-1 blocks than S", in other words, that n — a + 2 < b. By Lemma I4.1[ n — a + 2 > n — 2k' 
and b > n — 2k" . Then 

2b > (n - a + 2) + b > 2n - 2k' - 2A;", 

therefore b — a > n — 2 — 2k > n — 2 — (n — 2) = and 6 > n — k > n/2 + 1. Since 
d 2 < n/2 — 1 < 6 — 1, we can use Lemma [4.21 to adjust the first b columns in S" without 

(2) 

changing det(>S") (modulo lower degrees), so that z\j — Xi — Xj for 1 < i < j < b — 1. Note 

(2) 

that z ib can be either Xi — Xj or yi — yj for 1 < i < b. Similarly, we can adjust the last 
n — b + 2 columns in S' u such that = Xj — Xj for b < j < n and 1 < i < j ' . Then 

b— 1 j— 1 n j — 1 

det(^)=rnn^nn^ 

j=a i=l j=b i=l 

and 

det(Sl)det(S'l) = A J] ( Xi 

l<i<j<n 

One verifies that A is a polynomial of bidegree (di — (m — 1) (™) , o^) m ^- Applying the base 
case m = 1, we conclude that det(S^) det(S^) is a linear combination of 

< det(S^) Y\ ( x i ~ x i) \ 

I l<i<j<n J MePar ( d2)fc ) 

modulo lower degrees. This proves (*). □ 
Lemma 4.4. Let a be a positive integer. Then Yl^oPi^ ci — i) — p{a)- 

Proof. Given a partition v = (ui, . . . , vi) of a satisfying v\ < ■ ■ ■ < ue, we let i = i^, and send 
v to the transpose of the partition . . . , z^_i), which is a partition of a — i into at most i 
parts. This gives a one-to-one correspondence from Par(a) to |J" =Q Par(i,a — i). Counting 
the cardinalities of the two sets gives the stated equality. □ 

Corollary 4.5. Let n,m,k,d 2 G N satisfy n > 3, m > 0, and k + d 2 < n/2 — 1. Define 
d\ = mQ) — k — d 2 . Then the coefficient of q dl t d2 in AC ( /T\q,t) is 

divaM^=p{d 2 ,k). 

Proof. We recalled in the introduction that AC ( ™\q,l) = RCi m) (q,l) = V _ (m) q ^M. 

Then for each d\ y Ym 2 =o ( ^ ym ^df > d2 * s ^ e num ber of m-Dyck paths rr € with area(7r) = 
d\. Each such m-Dyck path uniquely determines a Ferrers diagram of size m(") — d\ consisting 
of the set of boxes above the m-Dyck path in the mn x n triangle. On the other hand, since 
any Ferrers diagram of size less than n determines an m-Dyck path and m(") — d\ = k + d 2 < 



6-1 j-1 \ /b-l 

(2) \ rpl, 



^•), det(^) = ( n n( Xl - x,) n z$ j r 

\j=2 i=l J \i=l 



b-l j-l \ (b-l 

(1) | TT J2) \ ^n// 



-x,), whereA = T'(nn^ II ^ ) ^ 

^j=a i=l / \i=l 
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each Ferrers diagram of size m(™) — d\ determines an m-Dyck path in Therefore 
the number of such m-Dyck paths is equal to the partition number p (m(™) — gM, and 



m(5)-di 



dim M^ 2 =p(m Q - <*i J = ~ dl ~ d: 

where the second equality is because of Lemma H~4l On the other hand, Theorem 14.31 asserts 
that 



di — d 2 ) ■ 



Therefore each inequality is actually an equality This implies dimM^ d = p(d 2 , k). □ 



Proof of Theorem \l.l[ For any fixed nonnegative integers k, h, whenever n > 2(k + h + 1), 

the coefficient of q m \^)~ h ~ h t h in AC^iq, t) is equal to p(h, k) by Corollary 14. 5[ The theorem 
follows from this observation and Propositions 13.21 and 13.31 □ 



5. Comparison of Coefficients of ACiT\q, t) to Partition Numbers. 
In this section, we extend the method in the previous section to obtain a theorem that 



identifies the dimensions of certain submodules M, 



(m) 



qdifd.2 j n ACn (q,t)) as partition numbers. 



di,d2 



(i.e., the coefficients of certain terms 



Theorem 5.1. Let n > 6 and m be positive integers, and let k, d\, d 2 be nonnegative integers 
such that k = m(2) — d\ — d 2 < n — 6. Let 5 = min(c?i, d 2 ). Then 



p(6,k). 



We need a few lemmas. 

Lemma 5.2 (Grafting Lemma). Let D\ = {Pi, . . . , P n } and D 2 = {Qi, . . . , Q n } be in D n , 

where the Pi andQi are listed in increasing graded lexicographic order. Suppose \P r \ = \Q r \ = 
r — 1. Let D[ = {Pi, . . . , P r _i, Q r , . . . , Q n } and D' 2 = {Q u . . . , Q r _i, P r , . . . , P n }. Then 



A(D 1 ) ■ A(D 2 ) = A(P>' 1 ) ■ A(D' 2 [ 



in . 



□ 



We omit the proof of the lemma. The following example illustrates the idea of the proof. 
Example 5.3. Consider P 1; D 2 , D[, and D' 2 pictured below. 



• •. mm 



Do 































K 




D' 2 = 
















( 


N 




t-t 




N 
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Then 



A(.Di) • A(D 2 ) = det 



and 



1 

x 2 i 

£312/32 x 3 ix 32 

£412/42 £41 £42 £41 £42 £43 

£512/52 x 51 x 52 0:51x52x53 



det 



1 

z/2i 

2/312/32 £312/32 £31^32 

2/412/42 £412/42 £41 £42 

2/512/52 £512/52 £5l£52 



A(Z) / 1 ) • A(D' 2 ) =det 



1 

£21 

2/312/32 £312/32 £3i£32 • det 

2/412/42 £412/42 £41 £42 

2/512/52 £512/52 £5l£52 

One readily verifies that the two products are equal. 



1 

y 21 

£312/32 £3l£32 

£412/42 £4l£42 £4l£42£43 

£512/52 £5l£52 £5l£52£53 

(2) 



Definition 5.4. For k < n — 4 and di + c?2 + & = 2 (™) , define a subspace iV^,^ of M d d by 



N, 



di ,d2 



M 



di,d2- 



4 m 



M 



-k,k 



d 2 —k 



if o?2 < A;; 
if di < k; 
otherwise. 



Lemma 5.5 (Higher Transfactor Lemma). Suppose k < n — A, d[ < d 1; d' 2 < d 2 , d\ + d 2 + k ■ 
2g) , and d[ + d! 2 + Q=d 1 + d 2 . 

(i) If d 2 < d 2 and d[ > k + 1, £/ien 

Md' lt d' 2 ■ fd 1 -d[,d 2 -d' 2 ^ ^4-1,4 + 1 ' /di-(^+l,(fa-(^-l 

as subspaces of M dd . 

(ii) i/d^ < c?i and d' 2 > + 1, taen 

Mii,^ ' fdi-d'^di-d'z Q Mii+1,4-1 ' /dx-t^-l^-t^+l 

as subspaces of M^ d2 . 

(iii) M d / ^ • f dl -d' x ,d 2 ~d' 2 is a subspace of N dl>d2 . Moreover, if d[,d 2 > k, then M^ jd » 
fd^-d' x ,d 2 -d' 2 is equal to N dud2 . 



Proof, (i) Let 



(n - 1, 0), if d' 2 < k; 

(n — l — d 2 + k,d 2 — k), if k < d' 2 < n - 2 + k; 

(l,n-2), " ifd 2 >n-2 + £;. 



Then there exists a basis {A(Di)} of M dl) d 2 such that the last point of each Di is P n . 
Indeed, consider the first case d 2 < k. Let ^'-0-1)4 ^ e the indicated graded piece of 
I n -i/vcin,-xln-i- Let {A(D' i )} be a basis of M' d , f n _-^ d i , and let Di be obtained from D[ by 
adding the point P n . Since M^„ d , and M dl ^ 2 have the same dimension p(d 2 , k), Lemma [2.61 
implies that {A(Di}} forms a basis of M dud2 . The other two cases can be proved similarly. 
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Now for each D { = {Pi, ... , P„}, define D' { = {Pi, ... , P„_i, P„ + (-1, 1)}. By the 
Grafting Lemma E21 we have A(A) • fd x -d' x ^-d' 2 = A(P-) ■ / dl -d' 1+ i,d 2 -4-i in AQ^. Then 
the inclusion stated in (i) follows immediately. 

(ii) This is symmetric to (i). 

(iii) This follows from (i) and (ii). □ 

Lemma 5.6. Assume n, cf 1; d 2 , A;', d", d 2 'j G N satisfy n > 6, k' = ("J — d[ — d' 2 , k" = 
g) - d'[ - <%, k' + k" <n- 6, and (d[, d' 2 ) + (d'{, d'£) = (d u d 2 ) ■ Then 



M d{A ■ M d » A C iV dlid2 



as subspaces of M d ^ d . 



Proof. Define n' = k' + 3. First, we claim that ^ has a basis consisting of elements of 
the form 

A(D') = A({P 1 / , . . . , P' n }), where |P/| = % - 1 for n' + 1 < % < n, 
and Md" )d » has a basis consisting of elements of the form 

A(D") = A({P 1 ", . . . , P^}), where |P/| = i - 1 for < i < ri. 

Indeed, one may find a pair of integers (e[, e 2 ) such that 

min(G?i, k') < e' x < d[ < e[ + P 



min(4, fc') < e' 2 < d' 2 < e' 2 + 



n \ In' 



2) V 2 
and e'i + e' 9 = ( „ ] — A;'. 



Then we choose P ? '/ + i, • • • , P' n such that |P/| = i — 1 for n' + 1 < i < n, and the sum of their 
bidegrees is (d[ — e[,d 2 — e 2 ). Choose a basis {A(P/)} of (7 n //m n // n /) e ' >e / and replace each 
P' = {Qi,...,Qn'}by 

-D' = {Ql; • • • ) Qn') Pn'+l; • • • > -^n}- 

In this way, we obtain a basis for Ms ^ • On the other hand, one can verify that there exist 
a pair of integers (e", e 2 ) and a nonnegative integer c < n' such that 



min«, k") < e'[ < d'[ - (n - n')c < e'/ 



2 

min(d 2 ', fc") < e 2 < d\ - (n - n')(n' - c) < e' 2 + 
and e" + e 2 = ( J — fc". 



2 



LIMITS OF MODIFIED HIGHER q, i-CATALAN NUMBERS 15 

Then we choose P", . . . ,P", such that \P!\ = i — 1 for 1 < % < n', and the sum of their 
bidegrees is (d'( — (n — n')c — e", d 2 ' — (n — n')(n' — c) — e'^). Take a basis {A(D")} of 
{I n -n' /vcin-n'In-n')e'{,e%, and replace each D" = {Q 1 , . . . , Q n _„/} by 

D" = {P", P",, Qi + (c, n - c), Q 2 + (c, n' - c), . . . , Q„_ n / + (c, n' - c)}. 
In this way, we obtain a basis for M d >i )d /'. 

Next, using the Grafting Lemma 15.2} 

A(D')A(D") = A({Pi, ...,P^,P:, +1 ,..., P'^)A({Pl . . . , P», P^, +1 , . . . , J*}), 
hence is in N dltd2 by Lemma l5.5( iii). □ 

Proof of Theorem \5.1[ Without loss of generality, we assume d\ > d 2 - After applying Lemma 
15.61 successively, we can conclude that 

for some nonnegative integers a, b, where a + b = [m — 1) Q) , and g a<b = Yli^i f<H,bi has 
bidegree (a, b). Moreover, by inspecting the proof of Lemma 15.61 carefully, we can assume 
b = max(0, d 2 — k). Therefore 

dimM^] 2 = dim(M dl _ a>d2 _ b ■ g a>b ) < dimM dl _ a42 _ b < p(d 2 , k). 

Now we prove dimM^] 2 > p(d 2 ,k). Take a sufficiently large integer n > n such that 
k, d 2 < n/2 — 1. Let M be In/main- Let 

h j-1 

fa = n ~ x ^- 

j=n+l i=l 

Define d\ — d\ + (n + h — l)(n — n)/2. By applying Lemma 12.61 successively, we conclude 
that the linear map h : M dl _ a4 , 2 - b —> M di _ ad b that sends / to / ■ fo is injective. Moreover, 
since k < n — 6, the domain and the codomain of h have the same dimension p d2i k- So h is 
also surjective. Consider the following commutative diagram: 

M dl _ a4 . 2 ^ h M- di _ a d2 _ h 



M dl _ a42 _ b ■ g a>b M di _ ad7 _ h ■ g atb 

i>2 



d\—a,d,2—b 

02 



where ~g af) = g a>b ■ (fo) m ~ l , tpi(f) = f • 0a,b L V>i(/) = f • 9a,b, and both the middle and bottom 
horizontal maps are given by /(->■/ • (fo) m . Since h and ipi are surjective and ip 2 is an 
isomorphism, the bottom horizontal map is surjective. By Corollary 14.51 

dimM^ 2 >dimM(^ 2 =p(rf 2 ,fc). 
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Thus the theorem is proved. 



□ 



In fact, we expect a stronger statement to hold: 

Conjecture 5.7. Let n > 2, m > 2, d±, c?2, k be positive integers such that k = ttiu) —d\ — d,2. 

Define S = min(<ii, g^)- Then dimM^] 2 < p(d~,k). Moreover, equality holds if and only if 
k < n - 2. 

6. Conjectures 

Conjecture 6.1. For tt G T>^ and 1 < i < mn, let o^tt) be the number of full squares 
in the i'th column below tt and above the line my = x, and let bi(n) be the number of full 
squares w in the i 'th column which are above tt and satisfy 

m ■ l(w) < a(w) < m(l(w) + 1). 
For tt G and 1 < j < m, let 

(*)),...,( 

a j+m(ra-l)( 7r ),b j+m{n ^ 1) (u))} cNxN. 

Then {riJLi det(-Dj(7r)) : tt G V^} generates the m-th power I™ of the ideal I n generated 
by alternating polynomials in C[xi, y±, . . . , x n , y n ]. 

Conjecture 6.2. Let I n be the ideal generated by alternating polynomials in R = C[x, y] = 
C[x\, yi, ... , x n , y n ] . Consider the minimal free resolution of I n : 

— > F n — > >F 1 ^R^ R/I n — > 0. 

Then for each 1 < i < n, the bigraded Hilbert series of 

TbTi{R/I n , C) = ToiiiR/I^R/m) 

is equal to 

i-iy- 1 £ (( Sl r,s x )(v(s x ),s iln) ). 

Ahn 
spin(A') = i — 1 

(Recall that {(si) n ,s\) = f x , the number of standard Young tableaux of shape X. For defini- 
tion of spin, see p. 6 in [T3J. ) 

The conjecture is verified for n < 6. As a special case, we have: 
Conjecture 6.3. The bigraded Hilbert series of I n is 

(l-^(l-t)" W ' S(1B)) ' 
Conjecture 16.31 follows from Conjecture 16.21 because = ^2w- n ((si) n , s\)s\. 
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7. Appendix: Comparison of Definitions of Higher q, ^-Catalan Numbers 

In the Introduction, we gave seven definitions (a)-(g) of the higher q, t-Catalan numbers. 
Here we explain the known relations among these definitions. 

(a) -^(b): There is an obvious bijection between partitions A G Par^™- 1 and m-Dyck words 
7 G , defined as follows. Given the partition A, embed the diagram of A in an mn x n 
triangle as shown in Figures [1] and [2j For < i < n, let 7, be the number of complete 
squares to the right of A and to the left of the diagonal in the (i + l)'th row from the 
bottom. For example, when m = 2, n = 5, and A = (7,5,4), we see from Figure [2] that the 
associated 2-Dyck word is 7 = (0, 2, 0, 1, 1). It is routine to verify that this process defines 
a bijection from Par^" - -' onto such that area c (A) = area(7). It is less routine to prove 
that c m (A) = dinv m (7); see {5J Lemma 6.3.3] for the proof. (Note that what we call c m (A) is 
called 6 m (A) in [5].) 

(b) <^(c): See [T2J §2.5] for a bijection from r„ to PA such that if 7 maps to it under 
the bijection, then area(7) = b m (7r) and dinv m (7) = area(7r). This proves WC { ^ n \q, t) = 
DCir\q } t). On the other hand, it is an open problem to define a bijection 7 h-» it from 

to T>n satisfying area(7) = area(7r) and dinv m (7) = b m (ir). This problem is equivalent to 
proving bijectively that the combinatorial definitions (a), (b), and (c) are symmetric in q 
and t. 

(d)-v^(e): One can use well-known facts about Macdonald polynomials to prove that 
SCiT\q,t) = RCi m \ qj t) (cf. [3] and 0). Indeed, since e n = £^J(l- g )(l-*)5 M n^/<)i^ 
and V(# M ) = TpHp, linearity of V gives V m (e„) = £^ n ((l - g)(l - t)T^B^/ Wfl )H^. 
Since {H^, e n ) = T M , we can conclude that (V m (e„), e n ) = X^hJ 1 -^T^B^/w^, 
as desired. 

(d)-v^(f): Let J be the ideal in C[x, y] generated by polarized power sums ^2i=i x i"yi 
{h + k > 1). One can also describe J as the ideal generated by all S^-invariant polynomials 
without constant term, where S n acts diagonally [7]. Let e be the sign representation of S n . 
It is proved in [3[ Proposition 6.1.1] that 

V m (e n (zi, Z2, ...)) = J 7 jm-i^/m-i J jim-i{zi, , . . . ; q , t), 

where the right side denotes the Frobenius series of e m ~ l <g> J m ~ 1 /JJ m_1 . (Note that the 
meanings of I and J are switched in [5|.) On the other hand, one may check that the S n - 
alternating part (e m_1 ®/ m_1 / J/ m_1 ) e is isomorphic to £ m_1 <g> I m /ml m . We can extract the 
S^-alternating part from the Frobenius series by taking the scalar product with e n = S(i«). 
Therefore, 

SC^(q,t) = (V m (e n ),e n ) = £ q u t v dim^" 1 " 1 <g> I m /mI m ) U)V 

U,U>0 

= ]T g M rdim(/ m /m/ m )^ = ^ gVdimA/J? = A<7< m >(g,*). 

■u,?;>0 u,v>0 
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(e)<^>(g): Haiman showed the identity 

n-l 

RCt\q,t) = ^(-l)W ( z„,OM)(g,t) 

in §3, Theorem 2]. Then he showed that for i > and I > 0, H\Z n , P®B® 1 ) = 0, where P 
and .B are the vector bundles defined in pm §2]. In particular, this implies H l (Z n , O(k)) = 
for i > pUl Introduction and Theorem 2.2]. Therefore RC ( fT\q,t) = tr H o^ Zn ^ ^ m ^(q,t), 
which is exactly GC^ n \q,t). 
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